Skip to content

[CI] Parity: count a disagreement only when CUDA PASSED#3371

Open
ethanwee1 wants to merge 1 commit into
developfrom
ethanwee/parity-cuda-passed-disagree
Open

[CI] Parity: count a disagreement only when CUDA PASSED#3371
ethanwee1 wants to merge 1 commit into
developfrom
ethanwee/parity-cuda-passed-disagree

Conversation

@ethanwee1

@ethanwee1 ethanwee1 commented Jun 24, 2026

Copy link
Copy Markdown

Summary

Changes the parity DISAGREE/AGREE metric to count a ROCm SKIPPED/MISSED test as a disagreement only when CUDA actually PASSED it (status_cuda == PASSED), instead of "CUDA didn't SKIP it" (!= SKIPPED).

Why

The old definition counted as disagreements the large set of attention-backend parametrization variants (e.g. test_transformers Flash/CK/CUTLASS cases) that CUDA never even enumerates (CUDA = MISSED). Those aren't real ROCm-vs-CUDA coverage gaps — they just inflated DISAGREE to ~3% (AGREE ~97%). Counting only "CUDA passed, ROCm didn't" reflects the actionable gap and matches the triage spreadsheet's adjusted definition.

Effect (mi350, real data)

  • Overall AGREE% 97.3% → 99.07%
  • DEFAULT SKIPPED+MISSED 9,002 → 2,629, DISAGREE 3.0% → 0.88%
  • The two line items are relabeled SKIPPED (on rocm, PASSED on cuda) / MISSED (on rocm, PASSED on cuda) for accuracy.

Scope

compute_test_config_stats + compute_overall_stats in generate_summary.py. The dashboard's parity collector (ROCm/AI-Frameworks-Dashboard#49) is being updated with the same definition so the dashboard matches this report.

Test plan

  • py_compile clean.
  • Ran against a real mi350 status CSV → Overall AGREE 99.07%, DEFAULT DISAGREE 0.88%.

Made with Cursor


Rebased onto current develop + validated

This PR was rebased onto current develop (it was previously opened against an old lineage and showed unrelated files). Validation parity run on the clean SHA ba9910c6 (mi300): https://github.com/ethanwee1/pytorch/actions/runs/28884929807

ethanwee1 added a commit to ethanwee1/pytorch that referenced this pull request Jun 24, 2026
Count a ROCm SKIPPED/MISSED test as a disagreement only when CUDA actually PASSED it (s2 == PASSED), excluding parametrization variants CUDA never runs. AGREE% becomes ~99%, matching the triage spreadsheet. Mirrors ROCm#3371.
@rocm-repo-management-api

rocm-repo-management-api Bot commented Jun 24, 2026

Copy link
Copy Markdown

Jenkins build for 7f5f023dd5f5dc4cfc6480a335e9392b33d762ec commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

jithunnair-amd pushed a commit that referenced this pull request Jun 26, 2026
#3375)

Two user-reported bugs in the parity report.

## 1. LOG-BASED FAILURES duplication
A test could be listed **twice** in the LOG-BASED FAILURES table — once
as `FAILED` and once as `CONSISTENT_FAILURE` — because
`detect_log_failures.py` emits both a per-test `FAILED` record and a
`FAILED CONSISTENTLY` record for the same test, and
`generate_summary.py` never deduped within `log_failures` (it only
filtered against the XML FAILED TESTS table).

**Fix:** add a shared `_select_rocm_log_failures()` helper (used by both
the CSV and markdown renderers) that filters out XML-failed tests and
**dedupes per test, preferring `CONSISTENT_FAILURE`** — so a test is
never shown as both.

Example that regressed: `mi350 · default · test_sparse ·
TestSparseMaskedReductionsCUDA ·
test_future_empty_dim_masked_prod_cuda_complex128` showed as both; it
now shows only `CONSISTENT_FAILURE`.

## 2. Empty Job ID columns in FAILED TESTS
`download_testlogs._shorten_unzipped_dirs` extracted the upstream CI job
id with `re.search(r'_(\d{6,})\.zip$', ...)`, but the **unzipped** shard
dirs end in `_<jobid>` with no `.zip` (e.g.
`unzipped-...gfx950.2_83371682957`). So the `_<jobid>` suffix was
dropped, dirs became `test-default-1-8`, and `summarize_xml`'s
`parse_xml_reports_as_dict` (`_(\d+)$`) couldn't recover the id ->
`job_url` was empty for every row -> the FAILED TESTS "Job ID" columns
were always blank.

**Fix:** make `.zip` optional: `r'_(\d{6,})(?:\.zip)?$'` (still matches
the older `.zip` form).

## Validation
- Both files `py_compile` / parse clean.
- Dedup: ran `_select_rocm_log_failures` on a real `log_failures` CSV —
the test that had both `FAILED` and `CONSISTENT_FAILURE` collapses to a
single `CONSISTENT_FAILURE` row (0 keys left with multiple categories).
- Job id: the new regex extracts `83371682957` from the real
`unzipped-...` dir name, and the resulting
`test-distributed-1-3_83371682957` parses correctly, so `job_url`
populates. End-to-end `Job ID` population will be confirmed by the next
parity run after merge.

Independent of #3370 / #3371 (those touch a different section of
`generate_summary.py`).

Made with [Cursor](https://cursor.com)

## Validation

Validated on a run of **this PR branch** (develop + both fixes) for the
exact SHA that exhibited both bugs:

- `206283a284889f3a4f62510f8b5a8068d41121c4 · mi350`:
https://github.com/ROCm/pytorch/actions/runs/28190615761
- **Bug 1:**
`test_sparse::TestSparseMaskedReductionsCUDA::test_future_empty_dim_masked_prod_cuda_complex128`
is now listed **once** as `CONSISTENT_FAILURE` (was both FAILED +
CONSISTENT_FAILURE); 0 keys with multiple categories across the
LOG-BASED table.
- **Bug 2:** FAILED TESTS table has **24 populated Job ID links**; in
the status CSV `job_url_rocm` went 0 -> 308,793 and `job_url_cuda` 0 ->
354,344 non-empty.

Also applied to `ethanwee1/pytorch@main` (`fe21d4e`); the fork run
(https://github.com/ethanwee1/pytorch/actions/runs/28189128743) confirms
Bug 1. Note: the fork's `summarize_xml_testreports.py` predates
develop's `_wf_run_id`->`job_url` feature, so Job IDs there populate
once the fork syncs that from `develop` (Bug 2's fix is the
`download_testlogs` regex, which is on the fork).
The DISAGREE/AGREE metric counted a ROCm SKIPPED or MISSED test as a
disagreement whenever CUDA merely did not SKIP it - which includes the
large set of attention-backend parametrization variants CUDA never even
enumerates (CUDA MISSED). Those are not real ROCm-vs-CUDA coverage gaps and
inflated DISAGREE to ~3% (AGREE ~97%).

Count a disagreement only when CUDA actually PASSED the test (s2 == PASSED)
in both compute_test_config_stats and compute_overall_stats, and relabel the
two line items accordingly ('PASSED on <set2>'). This matches the triage
spreadsheet's adjusted definition; AGREE% becomes ~99%.
@ethanwee1 ethanwee1 force-pushed the ethanwee/parity-cuda-passed-disagree branch from 7f5f023 to 9d939b6 Compare July 7, 2026 16:38
@rocm-repo-management-api

rocm-repo-management-api Bot commented Jul 7, 2026

Copy link
Copy Markdown

Jenkins build for 9d939b6f08e9c9f18f3896fa404afc7974d945d2 commit finished as ABORTED
Links: Pipeline Overview / Build artifacts / Test Results

Detected error during Pytorch building:

-- HIP architectures: gfx90a;gfx908;gfx942;gfx1033
CMake Warning (dev) at /opt/rocm/lib/cmake/hip/hip-config-amd.cmake:98 (message):
   GPU_TARGETS was not set, and system GPU detection was unsuccsessful.
   
   The amdgpu-arch tool failed:
   Error: 'Failed to get device count'
   Output: ''
   
   As a result, --offload-arch will not be set for subsequent
   compilations, and the default architecture
   (gfx906 for dynamic build / gfx942 for static build) will be used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant